Reduction of search ambiguity with multiple media references

ABSTRACT

Methods and apparatus for implementing a technique for searching media objects. In general, in one aspect, the technique includes receiving user input specifying a plurality of reference objects, defining a set of features for them, and combining the features to generate composite reference information defining criteria for search. In general, in another aspect, the technique includes combining object information for a plurality of reference objects to produce composite reference information, comparing the composite reference information to object information for media objects in a collection of media objects, and selecting a media object based upon the comparison.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.10/526,287, filed Mar. 1, 2005, which is the National Stage ofInternational Application No. PCT/US2002/031258, filed Sep. 30, 2002,each of which is hereby incorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

The present invention relates to searching media objects.

Electronic technologies for capturing, storing, and relaying digitalobjects such as images, audio files, and videos are now readilyavailable. Moreover, the increasingly common use of these technologieshas resulted in large numbers of readily obtainable media objects.People make pictures using digital cameras, drawing programs, andphoto-management software. They create video files with handheld videocams and burn CDs on home entertainment centers. The resulting digitalobjects are easily copied and stored, and can easily accumulate in anarchive. They are also easily shared on the World Wide Web, orInternet—for example, by email or on a website—making large numbers ofmedia objects available to many people.

As the number and accessibility of media objects increases, it canbecome increasingly difficult to manage them. For example, the larger aset of images becomes, the more difficult it can be to find a particularor desirable picture. A person may, for example, attempt to find aparticular image by recalling the time it was created or the directorywhere it was stored. Similarly, the greater the access to audio andvideo files, the more difficult it can be to find a particular ordesirable song or movie. A person may, for example, attempt to find aparticular audio or video recording by recalling its title, or the nameof its lead performer or actor. Using such information about the mediaobjects to define the search intent can be problematic, since it mayrequire a person to remember details about a particular object in orderto find it.

A person can search a collection of media objects for particularfeatures or aspects that are inherent to the object—not just associatedwith it. For example, a person can search for images that have aparticular distribution of colors, shapes, and textures by specifyingparameters describing these features. A person can attempt to search forimages that include, for example, a specific object or person byproviding a reference image and searching for images that are similar toit. Existing image search engines compare a selected reference image toimages in a database, and rank those images as more or less similar tothe reference. The process can be repeated, for example, using differentreferences to search smaller subsets of the database.

Information about a media object and information from a reference objectprovide useful bases to search a database. However, they impose limitson the criteria that can be used for a search, and often do notadequately characterize the searcher's intent.

SUMMARY OF THE INVENTION

The invention provides techniques for specifying search criteria bycombining reference features to reduce ambiguity in defining searchintent. In general, in one aspect, the invention provides methods andcomputer program products implementing techniques for combining objectinformation for a plurality of reference objects to produce compositereference information representing criteria for a search. The compositereference information is compared to object information for mediaobjects in a collection of media objects to identify one or more mediaobjects in the collection of media objects.

Advantageous implementations of the methods and computer programproducts can include one or more of the following features. A mediaobject in the collection of media objects can be selected based upon thecomparison of the object information and the composite referenceinformation. The plurality of reference objects can be specified by userinput. The plurality of reference objects can include one or moreobjects having a type selected from: audio, image, text, CD, video.Object information for different types of objects can be combined.Combining object information can include determining the intersection orthe union of the object information for the reference objects.

The object information can characterize features of the referenceobjects and the media objects in the collection of media objects. Thefeatures can be weighted to specify a relative importance of thefeatures. Weighting the features can include receiving user inputindicating the relative importance of the features. A feature can berepresented by the relative frequency of occurrence of each of severalvalues for the feature. The set of features can include colorinformation describing the relative frequency of occurrence of colors inan object. A feature for a first object type can be mapped to a featurefor a second object type.

The techniques can include combining object information for anadditional reference object with the composite reference information torevise the composite reference information. The additional reference canbe a media object identified by comparing the composite referenceinformation to object information for media objects. The revisedcomposite reference information can be compared to object informationfor media objects in the collection of media objects.

A similarity value, indicating the similarity of the object to thecomposite reference information, can be assigned to each of the mediaobjects in the collection of media objects. The similarity value of eachof the media objects in the collection of media objects can be less thanor equal to a similarity value calculated for each reference object. Themedia objects can be ranked according to their similarity values, and amedia object can be selected based upon its rank.

The object information for each of the reference and media objects canbe expressed as a feature vector of components, where each featurevector includes one or more components representing a feature of thecorresponding reference or media object. Each feature vector can includeone or more components representing metadata associated with thecorresponding reference or media object. The feature vectors of theplurality of reference objects can be combined to produce a compositereference vector. Components representing a feature of part or all ofeach reference object can be combined according to a first combinationfunction, and components representing metadata associated with part orall of each reference object can be combined according to a secondcombination function.

A weighting vector that specifies the relative importance of one or morefeatures can be defined and used in combining the feature vectors. A Minor Max function can be used to combine feature vectors. The compositereference vector can be compared to the feature vectors of each of theplurality of media objects in the collection of media objects. Thecomposite reference vector can be compared to the feature vectors ofeach of the media objects using a Min or Max function. Objectinformation for reference objects can be combined using a combinationfunction, and the composite reference information can be compared toobject information for media objects using a comparison function that isbased upon the combination function.

In one implementation, the object information can characterize featuresof the reference objects and the media objects in the collection ofmedia objects and be expressed as a feature vector of components; thefeature vectors of the plurality of reference objects can be combinedusing a Min or Max function to produce a composite reference vector, andthe composite reference vector can be compared to the feature vectors ofeach media object in the collection of media objects using a Min or Maxfunction; and a similarity value that indicates the similarity of thefeature vector of the media object to the composite reference vector canbe assigned to each media object in the collection of media objects,where the similarity value of each of the media objects in thecollection of media objects is less than or equal to a similarity valuecalculated for each reference object.

In general, in another aspect, the invention provides a system forsearching a collection of media objects. The system includes a means forcombining object information for a plurality of reference objects toproduce composite reference information representing criteria for asearch, and a means for comparing the composite reference information toobject information for media objects in a collection of media objects toidentify one or more media objects in the collection of media objects.

Advantageous implementations of the system for searching a collection ofmedia objects can include one or more of the following features. Thesystem can include a means for assigning a similarity value, indicatingthe similarity of the object to the composite reference information, toeach of the media objects in the collection of media objects, whereinthe similarity value of each of the media objects in the collection ofmedia objects is less than or equal to a similarity value calculated foreach reference object.

The object information can characterize features of the referenceobjects and the media objects in the collection of media objects and canbe expressed as a feature vector of components. The system can include ameans for combining the feature vectors of the plurality of referenceobjects to produce a composite reference vector, and a means forcomparing the composite reference vector to the feature vectors of eachof the media objects in the collection of media objects.

The invention can be implemented to realize one or more of the followingadvantages. A user can define search criteria that reflect the user'ssearch intent in conducting a search of a set of media objects. A usercan define search criteria that reflect the user's search intent evenwhen the intent is not clearly defined by a reference or by informationassociated with objects. The search criteria can be defined by selectinga set of reference objects. The search criteria can be definedautomatically given a selection of reference objects. Search criteriacan be defined as commonalities or the intersection among objects in aset of objects. Search criteria can be defined as inclusive or the unionof features in a set of objects. A user can refine the search criteriaaccording to the user's search intent. The search criteria can beredefined by adding an object to a set of reference objects. The searchcriteria can be weighted according to a combination of information for aset of images. A user can use different types of media objects to definethe search criteria. A user can search one type of media object usingsearch criteria defined by another type of media object.

The details of one or more implementations of the invention are setforth in the accompanying drawings and the description below. Otherfeatures and advantages of the invention will become apparent from thedescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 describes a method for searching using multiple referenceobjects.

FIG. 2 portrays the use of a method for searching using multiplereference objects by a user in a computer environment.

FIG. 3 shows a method for using multiple media objects to createcomposite reference information and search media objects.

FIG. 4 shows a method for using feature vectors of multiple mediaobjects to create a reference vector and search media objects.

FIG. 5 shows a feature vector for an image object and a feature vectorfor an audio object.

FIG. 6 shows a feature vector for a CD object, and a feature vector fora video object.

FIG. 7 shows the application of the Min and the Max functions to combineor compare a set of discrete components from two feature vectors.

FIG. 8 shows the application of the Min and the Max functions to combineor compare components from two feature vectors that approximatecontinuous functions.

FIG. 9 describes a search method that combines information for multiplereference objects and compares it to similar information for each ofseveral media objects.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

A collection of media objects can be searched to identify a particularobject or objects in the collection based on features of a referenceobject. The use of a reference object to search a collection of mediaobjects is ambiguous, at least in part, because the features of interestin the reference object are often ill-defined or inappropriately rankedin terms of their importance for the search. For example, it istypically unclear which features of a reference object are of interest.Even if features of interest are specified, the relative importance ofvarious aspects of the features, such as color, texture, and shape, istypically undefined. Moreover, the reference may not include features oraspects that are of interest to a user.

By using multiple references to characterize and refine search criteria,the ambiguity that is inherent in using a reference object to define thesearch criteria can be partially or wholly resolved. Multiple referenceobjects are used to create a composite reference and, if desired, aweighting of features. The composite reference and any weighting offeatures are defined by evaluating features of the selected referenceobjects—for example, by finding commonalities among or differencesbetween the selected reference objects. The composite reference and/orweighting can be further refined with the selection of additionalreference objects. In this way, the search criteria can be adjusted tobetter express or reflect a user's search intent.

FIG. 1 illustrates a method for searching a collection of objects basedon features of multiple reference objects. Two or more reference objectsare selected (step 2). The reference objects can be selected from thecollection of objects to be searched. Alternatively, the referenceobjects can be selected from other sources, such as a predefinedcollection of reference objects. The reference objects can also besupplied by the user, such as by importing a desired reference objectfrom an external source, or by creating the reference object, e.g.,using a drawing or painting program on a computer. The reference objectsare used to generate a composite reference (step 4). The compositereference can include, for example, information about features that arecommon to the reference objects or features that occur in any of thereference objects.

A collection or set of objects to be searched is identified (step 6).Starting with a first object in the collection (step 8), the object iscompared to the composite reference (step 10). The results of thecomparison, for example, a measure of similarity, are typically saved orstored (step 11). If there is another object (the YES branch of step 12)in the collection of objects to be searched, it is compared to thecomposite reference (steps 14 and 10). The method proceeds until thereare no more objects (the NO branch of step 12) in the collection ofobjects to be searched.

An object or objects in the collection of objects that satisfy searchcriteria defined at least in part by the composite reference can then beselected (step 16) based, for example, on the object's or objects'similarity or dissimilarity to the composite reference. Optionally, someor all of the objects in the collection can be ranked according to theirsimilarity or dissimilarity to the composite reference. The identifiedobject or objects, and the ranking, can be presented to a user.

As shown in FIG. 2, the search techniques described herein are typicallyimplemented in a computer environment. For example, a user 50 uses amonitor 52, speaker, or other device to receive information from acomputer 54. The computer can be, for example, a personal computer or aserver on a network. A collection 60 of media objects is maintained instorage on the computer 54, for example, in a file system or database.The user 50 selects two or more objects 70, 72, 74 for a search of thecollection 60. The objects can be from the collection or from anothersource. The objects can, for example, be images and the user may desireto find an image that shares certain features with the selected images70, 72, 74. A composite 76 of the selected objects is created. Objectsin the collection 60 are then compared to the composite 76, and one ormore images 80, 82 are selected. These images may be, for example, theobjects in the collection that are most similar to the composite. Theimages can then be displayed to the user 50 on the computer monitor 52.The process can be repeated. For example, the search can be refined byusing the composite reference 76 or one of the initial reference objects70, 72, 74 and one of the images 80, 82 identified in the search.

In general, the purpose of a search using a reference is to find objectsin a collection that are similar in some way to the reference. Thesearch is specified by the reference and a comparison function, which isused to assess the similarity of the reference to the objects beingsearched. The reference can be a single object or a composite reference.It includes at least some features or aspects that are of interest, butcan include features that are not of interest, as well. The comparisonfunction can specify weights that control the balance of features thatare considered in assessing similarity, and can be used to limit ortailor the search. For example, the comparison function can define thescope of the search, such as whether it is to be carried out globally orlocally, and it can specify criteria for similarity, such as the rangeof sizes or orientations that are to be considered similar. In thesearch, the similarity of the objects in the collection to the referenceis determined according to the comparison function. The objects can beranked as more or less similar to the reference according to theirsimilarity values so that a user can select, for example, the most orleast similar object, or a set of most or least similar objects.

A search that is defined by a reference and a comparison function mayfail to produce objects that satisfy a user's search intent, even ifsuch objects are available in the collection of objects being searched.The search can be focused or refined by combining information for anadditional reference with the previous reference or references. Thiscombination produces composite reference information if a singlereference was used previously, and redefines composite referenceinformation that was used previously. References can be combined, forexample, to encompass characteristics of any of them, or to specifycharacteristics that are common to all of them. The references also canbe used to define a weighting of features, and can be used to define anew comparison function. The composite reference information, theweighting, if any, and the new comparison function, if any, define newsearch criteria that form the basis for a new search.

A collection of media objects to be searched can include image objectsor files, as well as audio objects or files, text objects or files, andobjects or files that include combinations of media types such as image,audio, and text. Typically, such media objects will include digitaldata. Image objects can be searched for visual features such asparticular objects, color, texture, and shape. Audio objects can besearched for sound features, such as particular sounds or words, tone,amplitude, and loudness. A text object, such as an electronic book, canbe searched for language features such as particular words or phrases,sentence patterns, or grammatical structures.

Combination media can encompass, for example, audio, image, and textdata, and can be searched, for example, for audio, image, or textfeatures. For example, a video can be searched for features of aparticular image or for features of a person's voice. The video can besearched by treating each frame as a separate image and comparing eachframe to the desired images features. Also for example, a book thatincludes both text and images can be searched for images. The similarityof each image in the book to the image features can be determined. Thebook or video can be evaluated as more or less similar to the imagefeatures, for example, based on one or more of the similarity measuresof the images in it.

Media of different types can be combined to search media of the same ordifferent types. Such mixing of media requires that the media objectshave similar or analogous features. Different types of media objectscan, for example, have features that are not specific to the media type,such as pattern, variability, and periodicity. Such features can becombined and used to search objects irrespective of object type.Features of one type also can be translated or converted into featuresof another type.

As shown in FIG. 3, a collection or set of media objects 110 includestwo or more media objects 111-114. If there are N media objects, M_(a),in the set of media objects, then {M_(a)}={M₁, M₂, . . . M_(N)}.Similarly, a set of reference objects 120 includes one or more mediaobjects 121-123. If there are Z reference objects, R_(a), in the set ofreference objects, we have {R_(a)}={R₁, R₂, . . . R_(Z)}. In this case,Z references objects are combined to search N media objects.

Information for each of two or more reference objects 121-123 iscombined to create composite reference information, R_(c) 130. Theinformation can be combined, for example, according to a function of tworeference objects, g (R_(i), R_(j)) 132. If there are only two referenceobjects, for example R_(a) and R_(b) 121-122, information for each ofthe two reference objects is combined such that R_(c)=g (R_(a), R_(b)).If there are more than two reference objects, their object informationcan be combined in a pair wise sequence. For example, information forthe reference objects R_(a) and R_(b) 121-122 can be combined to, createcomposite reference information R_(ab) 25, such that R_(ab)=g (R_(a),R_(b)). The composite reference information R_(ab) 125 can then becombined with information about a third reference object, for exampleR_(Z) 123, to create the composite reference information R_(c) 130, suchthat R_(c)=g (R_(ab), R_(Z)).

Information from an additional object or from other combined objects canbe added to existing composite reference information, allowinginformation for many reference objects to be combined. More than onefunction can be used to combine information from more than two referenceobjects 121-123 in the set of reference objects 120. Object informationcan be combined by a weighted sum of reference object information. Ifthe reference images are ranked in importance, then the rank may serveas the weights. Object information for more than two reference objectscan be combined directly, for example, by summing, rather than by pairwise application of the function, g.

The composite reference information 130 is compared to information foreach of two or more media objects 111-114 in the collection or set ofmedia objects 110 using a function, G 134. The function G determinessimilarity, s 141-144, between the composite reference information andthe information for each media object. The similarity values 141-144 canbe used to rank the associated media objects 111-114 as more or lesssimilar to the composite information R_(c) 130 that was derived from thereference objects 121-123.

The function G 134 can include elements that are similar or identical tothe function, g 132, which is used to combine information for two ormore reference objects. For example, the element g_(s) (M₁, R_(c)), ofthe function G can be similar or dissimilar to the function g 132. Inthis way, a user can independently tailor the combination and comparisonfunctions according to the user's search intent. For example, a user cancombine references to create an inclusive reference, and then search forobjects that are more strictly similar to the reference. Also forexample, a user can choose a comparison function that complements thecombination function, for example, by using a comparison function thatbuilds on the combination function.

The function G can include a weighting element, W_(s) 150. The weightingelement can be derived from information for one or more of the referenceobjects 121-123, for example, by combining information about each of twoor more reference objects 121-123 according to a function, h (R_(i),R_(j)) 152. If there are only two reference objects, for example R_(a)and R_(b) 121-122, information for each of the two reference objects iscombined such that W_(s)=h (R_(a), R_(b)). If there are more than tworeference objects, their object information can be combined in a pairwise sequence as described previously, such that, for example, W_(s)=h(R_(ab), R_(Z)), where R_(ab)=h (R_(a), R_(b)). The function h istypically identical to the function g, in which case h (R_(a), R_(b))=g(R_(a), R_(b)) and R_(c)=W_(s).

Information that is used to characterize the features of media objectscan be summarized by a set of parameters. For example, informationcharacterizing a media object can be summarized as a series of values ina vector, referred to as a “feature vector.” A feature vector preferablyincludes information that is created or derived by analyzing part or allof a media object with one or more particular analytical methods. Afeature vector can also include information that is associated with theobject or the analysis of the object. Such information is commonlyreferred to as metadata.

An example of a search that uses feature vectors to combine informationfor multiple references and search a set of media objects is shown inFIG. 4. Each media object 111-114 in a set of media objects 110 can havea corresponding feature vector V_(M) 211-214. If there are N mediaobjects, M_(a), in the set of media objects, we have {M_(a)}={M₁, M₂, .. . M_(N)} and N corresponding feature vectors, V_(M1), V_(M2), . . .V_(M). Similarly, each media object 121-123 in a set of referenceobjects 120 has a corresponding feature vector, V_(R) 221-223. If thereare X reference objects, R_(a), in the set of reference objects, we have{R_(a)}={R₁, R₂, . . . R_(X)} and Z corresponding feature vectors,V_(R1), V_(R2), . . . V_(RZ). The feature vectors for the media andreference objects typically will have the same or similar components.

The feature vectors 221-223 corresponding to the set 120 of referenceobjects 121-123 are combined to create a composite reference vector 250.The feature vectors 221-223 can be combined, for example, according to afunction of two feature vectors, f_(c) (V_(i), V_(j)) 232, for example,in a pair wise fashion, as described previously for information aboutreference objects.

The composite reference vector 230 is compared to the feature vectors211-214 for each of two or more media objects 111-114 in the collectionof media objects 110 using a function such as Tr [W_(s)*f_(s) (V_(i),V_(r))] 234, where Tr indicates the sum of the components in the vector[W_(s)*f_(s) (V_(i), V_(r))]. If there are Z components, c, in each ofthe vectors W_(s), V_(i), and V_(r), then Tr [W_(s) f_(s) (V_(i),V_(r))]=Σ_((c=1 . . . z)) [W_(s)[c] f_(s)(V_(i)[c], V_(r)[c])]. Forexample, if [W_(s)*f_(s) (V_(i), V_(r))]=X and X has Z components, thenTr [W_(s)*f_(s) V_(i), V_(r))]=Tr X=X₁+X₂+X₃ . . . X_(Z). As for thefunction G 134, discussed previously, the function Tr [W_(s)*f_(s)(V_(i), V_(r))] 234 determines similarity, si 141-144, between thecomposite reference vector and a feature vector for a media object, i111-114. The similarity values can be used to rank the associated mediaobjects 111-114 as more or less similar to the composite referencevector 230 that was derived from the feature vectors 221-223 of thereference objects 121-123.

The function Tr [W_(s)*f_(s) (V_(i), V_(r))] 234 can include elementsthat are similar or identical to elements of the function, W_(c)*f_(c)(V_(i), V_(j)) 232. For example, it can be that f_(s) (V_(i),V_(r))=f_(c) (V_(i), V_(j)). The function Tr [W_(s)*f_(s) (V_(i),V_(r))] 234 also can include a weighting element, W_(s) 250. Theweighting element can be derived from the feature vectors 221-223 forone or more of the reference objects 121-123. A weighting element can bederived, for example, according to a function of two feature vectors,W_(w)*f_(w) (V_(i), V_(j)) 252, which can be applied in a pair wiserepetitive fashion as described previously to combine the featurevectors for multiple reference objects.

The function W_(w)*f_(w) (V_(i), V_(j)) 252 can include elements thatare similar or identical to elements of the functions W_(c)*f_(f)(V_(i), V_(j)) 232 or Tr [W_(s)*f_(s) (V_(i), V_(r))] 234. For example,it can be that f_(w)(V_(i), V_(j))=f_(f) (V_(i), V_(j)) or f_(w) (V_(i),V_(j))=f_(s) (V_(i), V_(j)). Similarly, it can be that W_(w)=W_(c) orW_(s)=W_(c).

The feature vectors for media objects, including reference mediaobjects, will now be discussed in more detail. A feature vector istypically a one-dimensional vector of components. For example, thefeature vector, V, is a series of W components such that V={V₁, V₂, . .. V_(W)}.

As shown in FIG. 5, a feature vector 300, 302; 350, 352 can include setsof components 310, 301-314; 321, 330-334. The components of a featurevector can be grouped into sets and subsets according, for example, tothe type of information of the components or the method by which thecomponents were created or derived. For example, the first 1000components of a feature vector may be derived from an analysis of themedia object and the last 3 components of the feature vector may beinformation associated with the media object. In this example, the first1000 components can, for example, be derived from determining theproportion of the image that is each of 1000 colors and the last 3components can, for example, be the date and time that the object wascreated and the filename of the object.

Components of some sets can be derived from the media object. Forexample, a feature vector for an image object can include sets ofcomponents describing texture, T 310, color, C 311, and shape, S 311,each of which are derived by analysis of some or all of the image data.Also for example, a feature vector 350, 352 for an audio object caninclude sets of components describing Fourier transform information F330, wavelet decomposition information, W 331, and fractal measures, Fr322, each of which are derived by analysis of some or all of the audiodata. A feature vector can include components that are derived from theanalysis of more than one type of media in the object. For example, afeature vector for an audio-visual object can include fractal measuresderived from the analysis of combined audio and visual parts or aspectsof the object.

Components of other sets can include information associated with theimage or audio object, or metadata. For example, set D 313, 330 caninclude the filename and the date and time that the image or audioobject was created, copied, or stored, and set M 314, 334 can includeinformation about the analyses used to derive the components in othersets.

The number of components in a feature vector 300, 350 is the sum of thenumber of components in each set of components in the feature vector.For example, if there are J, K, L, 2, and D components in sets T, C, S,A, and M, respectively, then the feature vector 300 has J+K+L+2+Dcomponents. That is, V_(i)={T_(i) (i=1 . . . J), C_(i)(i=J+1 . . . J+K),S_(i) (i=J+K+1 . . . J+K+L), A_(i) (i=J+K+L+1 . . . J+K+L+2), M_(i)(i=J+K+L+3 . . . J+K+L+2+D)}. Similarly, if there are U, V, W, 2, and Dcomponents in sets F, W, Fr, A, and M, respectively, then the featurevector 350 has U+V+2+D+W components.

Various numerical and statistical methods can be used to derive orcreate the components in a set in the feature vector of an image object.The methods that are used may depend upon the kind of data in theobject. For example, image objects may include raster graphics andraster data, or vector graphics and vector data. Vector data can be usedto create components that describe, for example, the number of objectsor strokes in the image, or the number of certain special effects in theimage.

Unlike vector data, raster data must be segmented or otherwise analyzedto identify objects or shapes in an image. Shapes can be defined, forexample, by determining regions of approximately constant color.Rectangular shapes can be found using the method described in U.S. Pat.No. 6,298,157, “Locating and aligning embedded images” which is herebyincorporated by reference in its entirety. A raster image can besegmented, for example, by flood-filling regions of similar color andtexture, imposing penalties for crossing edges.

Components in a shape set, S 312, of a raster or vector image object canbe created or derived by finding or defining shapes in the image andthen measuring shape characteristics for each shape. Shapecharacteristics can include measures of symmetry, ellipticity, orprolateness, for example. Shape characteristics can be defined, forexample, as the fractal dimension of the perimeter or, more simply, as alist of points along the curve of the perimeter of the shape or a listof tangent angles along the perimeter.

Components in the color set, C 311, of an image object can be created orderived by analyzing the image according to one or more color spaces. Acolor space provides a data representation for a range of colors interms of basic color components (or “colorants”). The specific colorantsdepend on the color system used. For example, in the CMYK color system,colors are represented as combinations of values for cyan (C), magenta(M), yellow (Y), and key (K) (generally black); in an RGB color system,colors are represented as combinations of values for red (R), green (G),and blue (B); and in the HSB color system, colors are represented ascombinations of values for hue (H), saturation (S) and brightness (B).

Color components can include, for example, measures of the mean andstandard deviation of colors in the image, or a list of the dominant ormost common colors. Color components can describe a frequencydistribution of colors, or the entropy of such a distribution. Colorcomponents can include the products of spatial coordinates with color.For example, components can be defined by <r C>, where r is the vectoror spatial dimensions and C is the vector of color dimensions. <r C> isdefined as 1/R τ_(r=1 . . . R) (r_(i) C_(a)(r)), where there are Rlocations, r, and where r_(i) is one of the spatial coordinates at r andC_(a)(r) is one of the color dimensions for the color at r. Suchproducts can include higher spatial frequencies, for example, <r rC>=1/R Σ_(r=1 . . . R) (r_(i) r_(j) C_(a)(r)) or <r r r C>. Componentscan include the mean and standard deviation for the products of spatialcoordinates with color.

In a frequency distribution of colors, for example, each of thecomponents in the set C 311, p(C_(i)), can represent the frequency ofoccurrence in the image of one of K colors. That is, each component candescribe the fractional area of the image covered by a color, C_(i). Forexample, an image can be divided into five colors such that one-fifth ofthe image is one color, one-tenth is another color, three-tenths is athird, three-tenths is a fourth color, and one-tenth is a fifth color.The set C for such an image is {0.2, 0.1, 0.3, 0.3, 0.1}. The set C canbe viewed as a frequency plot 500, as shown in FIG. 7.

For raster data, an image is made of pixels and each pixel, r, is of acertain color, C(r). The color is typically represented by a series ofbits (the “color value”), with specific bits indicating the amount ofeach colorant used in the color, as discussed previously. The valuep(C_(i)) can be calculated as the number of pixels of color i divided bythe total number of pixels, R. That is:

p(C _(i))=1/R Σ _(r=1 . . . R)Δ(C _(i) ,C(r)),

where

Δ (C_(i), C(r))=1 if C(r) found to be the same as the color C_(i), and

-   -   0 if C(r) not found to be the same as the color C_(i).        In this example, the sum of the p(C_(i)) over all possible        colors in an image is unity:

Σ_(i=1 . . . K) p(C _(i))=1.

However, functions that do not have the property of summing to unity arepossible and can be used as well. For example, the functionp(C_(i))=Σ_(r=1 . . . R) Δ (C_(i), C(r)), for which Σ_(i=1 . . . K)p(C_(i))=R, can be used. Also for example, an incomplete set ofproperties can be used. For example, a color space may be divided into Kcolors but information for only some, H, of those colors may be includedin the feature vector. That is, if Σ_(i=1 . . . K) p(C_(i))=1, thefeature vector can include only C_(i) where i=1 . . . H and H<K, suchthat Σ_(i=1 . . . H) p(C_(i))<1.

For vector data, a similar color distribution table can be produced byother means. For example, a grid can be mapped to the data, and eachcell in the grid can then be treated as a pixel would be treated tocalculate the color distribution. Also for example, the image can bedivided or flattened into constant color regions, and entries can beplaced in the color distribution table in proportion to the size of theregions of corresponding color. In yet another example, a colordistribution can be produced from the distribution of line and fillcolors of individual vector objects in the image object.

Components in a texture set, T 310, of a raster or vector image objectcan be created or derived, for example, by calculating the statisticalmoments of Gabor filter values for the image data. Such an analysiscaptures edge information in the image. Texture components can befractal measures of the image. Texture components also can becoefficients for a wavelet decomposition or Fourier transform of theimage data, or the means and standard deviations of the coefficients.Correlations between color components at two and more locations can beused as texture components.

Numerical and statistical methods, including those used to derive orcreate the components in a feature vector of an image object, can beused to derive or create the components in a feature vector 320 of anaudio object. Features can be calculated for the audio track as a whole,or for each of two or more segments of the track. The components of F330 can, for example, be derived from a Fourier transform of the audiodata. The components of W 331 can, for example, be derived from awavelet decomposition of the audio data. The components of Fr 332 can,for example, be the fractal dimension of the data, or the standarddeviation of the fractal dimension. Components in the feature vector ofan audio object can include a power distribution of audio frequencies,analogous to the distribution of color discussed previously.

As shown in FIG. 6, a feature vector 400, 401 for a media object thathas both image and audio information, such as a compact disc or CD, caninclude a set of image components 403 and a set of audio components 404,as well as components that describe the object in its entirety, asdescribed previously. The image components can include components andsets of components 310-314 as for an image object, as discussedpreviously, and the audio components can include components and sets ofcomponents 330-334 as for an audio object, as discussed previously.

A feature vector 420, 422 for a video object can include multiple setsof image components 423 and audio components 424. For example, thefeature vector can include a set of image components for key frames inthe video. There can be components that describe temporal relationshipsamong images, such as the number of scene changes and their duration. Ameasure of optical flow can be used to describe the amount of motion inthe video; this measure can be used, for example, to distinguishsurveillance tapes with activity from those without activity. A featurevector for a video can also include components 433 derived from part orall of the object, as discussed previously, and metadata 434 for part orall of the object.

Similar methods can be used to derive or create the feature vectorcomponents of different types of media objects or different segments ofa combination media object. For example, a cepstral decomposition, inwhich, for example, Fourier or wavelet frequency components taken over alocal time window are plotted as a function of time (that is, locationin the audio track), can be used to create a two-dimensional “image” ofthe audio signal. Such a two-dimensional audio object can then bedescribed with components as for an image object. For example, an audiotexture feature can be produced by wavelet decomposition of thetwo-dimensional audio object or segment, while an image texture featurecan be produced by wavelet decomposition of an image object or segment.The audio object's temporal frequencies then can be mapped onto theimage's spatial frequencies. With an appropriate choice of scales,features created for audio and image objects or segments can be combinedand compared.

A feature vector for a text object can include components thatcharacterize the whole text or parts of it, for example, chapters orparagraphs. The components can be derived from statistical measures ofparts of the text, such as the co-occurrence probabilities of semanticconcepts or words, using, for example, bigrams, trigrams, and so on.Text can be mapped to a semantic tree that is used as the feature.Components in the feature vector of a text object can include adistribution of frequencies of words or other constructs, analogous tothe distribution of color discussed previously. Methods similar to theseand others described previously can be used to derive or createcomponents for feature vectors of other types of media.

The functions 232, 234 for combining and comparing feature vectors willnow be discussed in more detail. These functions 232, 234 typicallyinclude a function, f, of two feature vectors, but can include afunction of more than two feature vectors. The function, f, typicallyevaluates one or more components of the feature vectors. Typically, thefunction does not find a Euclidian distance between two or more vectors.The function represents a combination of the features described by thefeature vectors. For example, the function can approximate theintersection or union of features. The function can approximate othernon-Euclidean combinations of features, as well. Two examples of such afunction will now be discussed in more detail.

As shown in FIG. 7, the components of two feature vectors 500, 510 canbe combined or compared, for example, by determining their jointminimum, Min 520, or joint maximum, Max 530. In this example, thecomponents in the set C of the feature vectors V_(i) 500 and V₂ 510 arescaled to range from 0 to 1, as discussed previously. For V_(i) 500,C={0.2, 0.1, 0.3, 0.3, 0.1}, indicating that one-fifth of the firstimage is C₁, one-tenth is C₂, three-tenths is C₃, three tenths is C₄,and one-tenth is C₅. The sum of the components is 1.0 because all colorsin the first image were tabulated. For V₂ 510, C={0.2, 0.1, 0.3, 0.1,0.3}, indicating that one-fifth of the second image is C1, one-tenth isC2, three-tenths is C3, one-tenth is C4, and three-tenths is C5. The sumof the components is 1.0 because all colors in the second image weretabulated.

A Min function determines the intersection of the sets or functionsdefined by two vectors. The vectors V₁ and V₂ are combined or comparedusing a “Min” 520 function by determining, for each component, thesmaller of the values for the two vectors. For example, the value of C₄for V_(i) is 0.3 and the value of C₄ for V₂ is 0.1. The smaller value,0.1, is represented in the resulting combination or comparison vector,V_(N) 521. In this example, the combination or comparison vector,V_(min)=V_(N), has C={0.2, 0.1, 0.3, 0.1, 0.1}. For a very large set,the components of C may approximate a continuous function 600, 610, asshown in FIG. 8. The application of a Min function 620 to two offset andapproximately normal functions 600, 610 produces a peaked curve 621,which represents the region of overlap of the two functions.

A Max function determines the union of the sets or functions defined bytwo vectors. The vectors V₁ and V₂ are combined or compared using a“Max” 530 function by determining, for each component, the larger of thevalues for the two vectors. For example, the value of C₅ for V₁ is 0.1and the value of C₅ for V₂ is 0.1. The larger value, 0.3, is representedin the resulting combination or comparison vector 531. In this example,the combination or comparison vector, V_(max)=V_(X), has C={0.2, 0.1,0.3, 0.3, 0.3}. For a very large set of components that approximate acontinuous function 600, 610, as shown in FIG. 8, the application of aMax function 630 to two offset and approximately normal functions 600,610 produces a bimodal or flattened curve 631. The bimodal or flattenedcurve represents the combined area under the two functions.

The Min or Max functions can be applied to feature vectors or sets infeature vectors that have continuously valued components. That is, theuse of such functions produces meaningful results if the strength ofeach component to which it is applied increases or decreasesmonotonically with the value of the component. Any probabilitydistribution satisfies this criterion, such that the application of theMin or Max function produces meaningful results.

Other components may satisfy this criteria. For example, the date andtime that an object was created can be included in its feature vector asa single extra component. If the value of the component is the time fromsome origin time, then the feature component satisfies the criterion.The combination of two such feature components with the Min functionwill produce the earlier creation time, whereas the combination with theMax function produces the later creation time. Also for example, thedominant image color can be included in a feature vector as a singlevalue, for example, the sum of the color's red, green, and bluecomponents. The minimum of two dominant colors of different intensitiesis a similar color with reduced intensity, which satisfies thecriterion.

Most continuously valued components can be translated into a series ofcomponents for which application of a function such as Min or Maxsatisfies the criterion. That is, most features can be mapped to adistribution function by binning. For example, the creation time of anobject can be expressed as a series of bins, such as one for each day ofthe year, p(day), and one for each of a number of years, p(year). If theobject was created on day=10 and year=20, then p(10)=1, p(20)=1, and allother p(day) and p(year) values are zero. Similarly and for example, thepotential values for a texture measure, t, can be divided into bins. Ift ranges from 0 to 1, the bins can be defined, for example, as follows:

bin range of values of t 1 0.0 to <0.1 2 0.1 to <0.3 3 0.3 to <0.7 4 0.7to <0.9 5 0.9 to 1.0 

Each bin is a component of the feature vector, T, and can take the valueof 1 or 0 such that, for example, if t=0.3, then T={0, 0, 1, 0, 0}. Thistechnique works for any feature whose value has a finite range or forany set of values. An component whose value has an infinite or verylarge range can be translated into a smaller finite range. For example,the number of words in a document, n, might range from 0 to millions,but can be translated as n′=log(n+1), which has a smaller and finiterange.

A Min or Max function can be used to combine or compare features ofdifferent types of objects. For example, features that are not specificto the type of media can be combined and compared to objectsirrespective of object type as discussed above. Features that arespecific to one type of media can be translated or converted intofeatures of another type and then combined with or compared to objectsof the latter type. For example, the frequency distribution ofwavelengths for an audio object can be mapped to a frequencydistribution of colors by mapping audio wavelengths to colorwavelengths. A proportionality constant or function can be used toconvert one to the other so that, for example, the distributions havesimilar ranges or so that the distributions' for particular audio andimage data have similar shapes. If the audio and color wavelengthdistributions have similar bin structure, they can be compared orcombined in a meaningful way.

The function that is used to evaluate the similarity between the objectsbeing searched and the reference, G [W_(s), g_(s)(M_(i), R_(c))] 134,can be tailored to complement the function that is used to combine theobjects and produce the reference, as discussed previously. For example,the similarity measure can be based on the comparison function. Afunction such as Min (V₁, V₂) or Max (V₁, V₂) defines a combinationvector such as V_(min) or V_(max), respectively. The sum or trace, Tr,of the components in a combination or comparison vector such as V_(min)can be an indication of the similarity, s, between the vectors beingrelated, V_(i) and V_(j). Thus, the similarity of the vectors i and jcan be defined as the trace of a combination function as applied tothose vectors. That is,

$\begin{matrix}{{s\left( {V_{i},V_{j}} \right)} = {{Trf}\left( {V_{i},V_{r}} \right)}} \\{= {TrV}_{\min}}\end{matrix}\quad$

For example, the components of the vectors V₁ 500 and V₂ 510 each sum tounity, but the components of the vector V_(N) 521 sum to 0.8, such thats (V₁, V₂)=Tr V_(N)=0.8. This measure indicates that the vectors V₁ andV₂ are 80% similar.

The function, f, is typically chosen to return a vector such that thesum of its components is between 0 and 1 inclusively, where 0 indicatesno similarity and 1 indicates complete similarity, i.e. identity. Ingeneral, if a set of components in two feature vectors is standardizedsuch that the sum of the components is unity, a vector derived by theapplication of the Min function will have such properties. If theobjects are identical in the attributes characterized by the set ofcomponents in the feature vector, then the components of V_(min) sum tounity. For example, Min (V₁, V₁) is V₁, and the sum of the components inV₁ is 1, indicating that the vector V₁ is completely similar—i.e.identical—to itself. If the two objects are not similar in any way thatis characterized by the set of components in the feature vector, thenthe components of V_(min) sum to zero. Thus, the Min function producesvectors for which the sum of the components ranges from 0 to 1 whenapplied to standardized sets of components, depending upon thesimilarity of the objects.

If feature vectors are not standardized to sum to unity, the Minfunction can be adjusted to produce vectors for which the sum of thecomponents ranges from 0 to 1 as follows:

s(V _(i) ,V _(j))=Tr[Min(V _(i) ,V _(j))]/Min(Tr V _(i) ,Tr V _(j))

This particular form is chosen in order to produce the followingdesirable property: A reference vector, V_(r), that is produced bycombining two vectors, V₁ and V₂, is defined as totally similar to eachof V₁ and V₂. If, for example, the composite reference vector, V_(r), is

V _(r)=Min(V ₁ ,V ₂),

then its similarity to V₁ is

s(V _(r) ,V ₁)=Tr(Min(V _(r) ,V ₁)]/Min(Tr V _(r) ,Tr V ₁).

which reduces to

s(V _(r) ,V ₁)=Tr[Min(V ₁ ,V ₂)]/Tr[Min(V ₁ ,V ₂)]=1.

Likewise, the similarity between the composite reference vector, V_(r),and V₂ is unity,

s(V _(r) ,V ₁)=Tr[Min(V ₁ ,V ₂)]/Tr[Min(V ₁ ,V ₂)]=1.

This adjustment allows one, for example, to compare the vector V_(N)521, where C={0.2, 0.1, 0.3, 0.1, 0.1} and Tr V_(N)=0.8, with the vectorV_(i) 500, where C={0.2, 0.1, 0.3, 0.3, 0.1} and Tr V₁=1.0. In thiscase, Min (V_(N), V₁)={0.2, 0.1, 0.3, 0.1, 0.1} and Tr [Min (V_(N),V₁)]=0.8.After adjustment, we have

s(V ₁ ,V ₂)=0.8/Min[0.8,1.0]=1,

indicating that the combination vector, V_(N), is completely similar tothe vector V₁. In general, the adjustment feature has the usefulproperty that a vector produced by application of the Min function iscompletely similar, after adjustment, to either of the two vectors fromwhich it was derived. That is

s(V _(i) ,V _(min))=s(V _(j) ,V _(min))=1

This identity also holds if the two feature vectors, V_(i) and V_(j),are combined with the Max function rather than the Min function. Thatis,

s(V _(i) ,V _(max))=s(V _(j) ,V _(max))=1

This important property ensures that reference objects used to definethe search criteria are identified as similar to the reference when theythemselves, or very similar media objects, occur in the collection ofobjects being searched. That is, each reference item, if included in thecollection of objects that is being searched, will be ranked as the mostsimilar of the objects to the composite reference.

In comparing two feature vectors, various features or components can beemphasized or de-emphasized by weighting them more or less relative toone another. For example, the first three of the five components in theset of components, C, may be of interest whereas the last two are not.In this case, the components can be weighted so that only the firstthree of them are considered. This particular weighting is accomplishedby multiplying the first three components of the vector by 1 and thelast two components by 0. In general, the combination or comparisonfeature vector can be multiplied by a weighting vector, W_(s) 50, togive, for example, W_(s)*Min V_(i), V_(j)) or, equivalently,W_(s)*V_(min).

The use of a weighting vector maintains the similarity propertiesdiscussed previously. Using a weighting vector, the similarity of twovectors, V_(i) and V_(j), whose components are normalized so that theirweighted sum is unity, can be defined as:

$\begin{matrix}{{s\left( {V_{i},V_{j}} \right)} = {{Tr}\; {W_{s}}^{\star}{f\left( {V_{i},V_{r}} \right)}}} \\{= {{Tr}\; {W_{s}}^{\star}V_{\min}}}\end{matrix}\quad$

More generally and for non-standardized sets of components:

s(V _(i) ,V _(r))=Tr[W _(s)Min(V _(i) ,V _(r))]/Min(Tr[W _(s) V _(i)],Tr[W _(s) V _(r)]).

These functions maintain the useful property that a vector produced byapplication of the Min function is identically similar to either of thetwo vectors from which it was derived, as discussed previously.

The weighting vector, W_(s) 50, 250 can be derived from information forone or more of the reference objects 121-123 by combining informationabout each of two or more reference objects according to a function, h(R_(i), R_(j)) 52, as discussed previously. If the information isrepresented as feature vectors, the weighting vector can be derivedaccording to a function h (R_(i), R_(j))=W_(w)*f_(w) (V_(i), V_(j)),where, for example, f_(w) (V_(i), V_(j))=Min (V_(i), V_(j)) or f_(w)(V_(i), V_(j))=Max (V_(i), V_(j)) and W_(w) is defined, for example,according to the user's interests.

Data or parameters that cannot be binned or meaningfully compared usinga function such as Min or Max can be included in a feature vector buthandled differently from distribution data when combining or comparingfeatures. For example, a feature vector, V, can include a distributionset, P={p(x₁), p(x₂), p(x₃), . . . p(X_(M))}, and other data, D={y₁, y₂,y₃, . . . y_(N)}, such that V={P, D}. Two feature vectors, V₁ and V₂,can be combined using two functions, one that is applied to P andanother that is applied to D. For example, the vectors can be combinedby applying the Min function to the sets of components, P₁ and P₂, whiletaking the average of the components in D₁ and D₂:

V _(r)={Min(P ₁ ,P ₂),{D1+D2)/2}.

The resulting reference vector, V_(r), can them be compared to thevectors of other media using two functions. For example,

s(V _(i) ,V _(r))=s(P _(i) ,P _(r))+K*abs(D _(i) +D _(r)),

where K is a constant that controls the importance of the parameters inD to the similarity comparison and s(P_(i), P_(r)) is the similaritymeasure used for distributions, Tr [W Min(P₁, P₂)]/Min (Tr W P₁, Tr WP₂). If x ranges from 0 to 1, K typically is a number between 0 and 10;for example, 1.0.

FIG. 9 illustrates a method for searching a collection of media objectsusing combined information for each of two or more reference objects ina set of reference objects. A user selects (step 701) a plurality ofreference objects to define the search criteria. Typically, the userdesires to find media objects that are similar to the selected referenceobjects. The selected reference objects may have particular features ormay have a general aspect that the user would like to find in othermedia objects. The selected reference objects may or may not sharefeatures or aspects that the user would like to find in other mediaobjects. The user also selects (step 702) a collection of media objectsto be searched. The user typically desires to find objects in theselected collection of media objects that are similar to the selectedreference objects.

The method requires information (step 704) about each reference objectand the media objects to be searched. Such information can beprecalculated and stored so that it is available for a search (the YESbranch of step 704). For example, the components for feature vectors canbe defined and the feature vectors for each reference object and each ofthe media objects to be searched can be derived before starting asearch. The feature vector for each object can be stored in associationwith the object.

The user can define and compute information for the reference object orthe media objects to be searched (step 706) if such information is notavailable or if the user prefers not to use the available information(the NO branch of step 704). For example, the user can specify whichsets of components and which components should be included in thefeature vectors. The feature vectors for the selected reference objectsand media objects can then be calculated. Typically, the information foreach of the reference objects and each of the media objects to besearched will be similarly defined. For example, the feature vectors ofthe reference and media objects to be searched will have the same seriesof components. The information for the objects can be dissimilar ornon-overlapping, in which case the information that is used in thesearch is typically limited to information that is shared among thereference objects and the media objects to be searched.

The method requires a combination function g (R_(i), R_(j)) 32, forexample, W_(c)*f_(c)(V_(i), V_(j)) 232, for combining the selectedreference objects (step 708). Part or all of the combination functioncan be predefined so that it is available for a search (the YES branchof step 708). The user can define a combination function (step 710) ifsuch information is not available or if the user prefers not to use theavailable information (the NO branch of step 708). For example, thefunction, f_(c) (V_(i), V_(j)), can be predefined while the user definesthe weighting vector, W_(c), for example, according to the relativeimportance of features to the search. The user also can define thefunction, f_(c) (V_(i), V_(j)), according to the intent of the search.For example, the Max function satisfies the intent of searching for thecombined set of features in the reference objects, whereas the Minfunction satisfies the intent of searching for those features that arecommon to all the reference objects.

The method also requires a comparison function G [W_(s), g_(s) (M_(i),R_(c))] 34, for example, Tr [W_(s)*(V_(i), V_(r))] 234 (step 712). Partor all of the combination function can be predefined, or calculatedaccording to predefined methods (the YES branch of step 712). Forexample, W_(s) may be omitted or predefined to be a uniform weightingvector, such that no features are weighted more or less than any others.W_(s) can be predefined to be the same as the weighting function, W_(c),used to compare reference objects. Alternatively, W_(s) can becalculated according to a function h (R_(i), R_(j)) 52, for example,W_(w)*f_(w) (V_(i), V_(j)) 252, and the function, h (R_(i), R_(j)), canbe previously defined. For example, it can be defined to be the same asthe defined combination function g (R_(i), R_(j)) 132 or W_(c)*f_(c)(V_(i), V_(j)) 232.

The user can define W_(s) (step 714) if it is not predefined or if theuser prefers not to use the predefined W_(s). For example, the user canspecify a weighting vector, for example, by specifying a type of filterfunction. Alternatively, the user can define a function f_(w) (V_(i),V_(j)) and a vector W_(w) as discussed previously for the definition ofthe combination function.

The use of different functions f (V_(i), V_(j)) to derive the weightvector for searching, W_(s), and the combination reference vector,V_(r), provides the user with more control over the results of thesearch than when the same function is used for both purposes. Forexample, the user can combine features using the Max function, f_(c)(V_(i), V_(j))=Max (V_(i), V_(j)) so that V_(r)=V_(max). Then, incomparing the reference vector to the feature vectors of the mediaobjects, the user can weight the features according to the Min function,f_(w) (V_(i), V_(j))=Max (V_(i), V_(j)) so that W_(s)=V_(min). In thisway the user encompasses all the features in the reference objects inthe search, but emphasizes those features that are shared between thereference objects when conducting the search.

As for the weighting vector, W_(s), the function, f_(s) (V_(i), V_(j)),can be predefined or automatically redefined (the YES branch of step712), or newly defined by the user (the NO branch of step 712). Thefunction, f_(s) (V_(i), V_(r)), can be predefined, for example, as Min(V_(i), V_(r)). The function f_(s) (V_(i), V_(r)) can be automaticallyredefined, for example, to be the same as the function f_(c) (V_(i),V_(r)) used to combined the reference objects. Alternatively, thefunction f_(s) (V_(i), V_(r)) can be newly defined by the user accordingto the user's search intent, as discussed previously for the definitionof the combination function.

The information for the reference objects is combined (step 716)according to the defined combination function and weighting, if any, toproduce composite reference information R_(c) 30, for example, V_(r)230. The user can then choose whether to conduct a search (step 718) ofthe previously selected media objects using the composite information.If the search is conducted (the YES branch of step 718), the compositereference information is compared to the information for each of themedia objects in the previously selected collection of media objectsusing the previously defined comparison function (step 720). For eachcomparison, a similarity value can be determined. The similarity valuescan then be used to identify media objects that are more or less similarto the composite reference information.

If a search is not conducted (the NO branch of step 718) or after thesearch is completed (step 720), the user can choose whether to add areference object to the set of reference objects (step 722). If not (theNO branch of step 722), the process ends. However, the user may, forexample, want to adjust the search to incorporate features or aspects ofadditional objects, such as objects identified in a previous search. Ifthe user so desires (the YES branch of step 722), the user selects anadditional reference object (step 724).

If the newly selected reference object is not one of the objects in thepreviously selected collection of media objects (the NO branch of step726), it may be necessary to calculate the required information (step704). If so (the YES branch of step 704), the user preferably creates orderives information for the newly selected reference object as for thepreviously selected reference objects. If the newly selected referenceobject is from the previously selected collection of media objects (theNO branch of step 726), the required information will have already beencreated or derived.

The user can next choose whether to newly define the combinationfunction (step 708). For example, the user may have created an initialcomposite vector using the Min function, but may wish to add a thirdvector using the Max function. In this way, the user can incorporateunique features of the newly selected object into the existing compositeinformation.

The user can also choose whether to newly define the comparison function(step 712). For example, the user can redefine the function, f_(s). Moretypically, the user may wish to redefine the weighting vector. Thecomparison function can change as multiple media objects are combined inthe selection process. For example, the comparison function will changeif the weighting vector changes. The weighting vector will usuallychange with the addition of information from newly selected referenceobjects if it is defined as a combination of the reference vectors. Inone implementation, the comparison function does not change as theinformation for multiple reference objects is combined—only thecomposite reference information changes. If this is not the case, theuser may choose to maintain previously existing weighting vectors.Alternatively, the user may want to emphasize the unique features of anewly selected reference object by defining a new weighting vector.

The information for the newly selected reference object is combined withthe existing composite information to create new composite information(step 716) and new weighting vectors, if necessary and desired. Themethod then proceeds as described previously. If a new search isconducted (the YES branch of step 718) the newly defined compositeinformation, the new weighting vector, if any, and the newly definedcomparison function, if any, are used.

The invention can be implemented in digital electronic circuitry, or incomputer hardware, firmware, software, or in combinations of them.Apparatus of the invention can be implemented in a computer programproduct tangibly embodied in a machine-readable storage device forexecution by a programmable processor; and method steps of the inventioncan be performed by a programmable processor executing a program ofinstructions to perform functions of the invention by operating on inputdata and generating output. The invention can be implementedadvantageously in one or more computer programs that are executable on aprogrammable system including at least one programmable processorcoupled to receive data and instructions from, and to transmit data andinstructions to, a data storage system, at least one input device, andat least one output device. Each computer program can be implemented ina high-level procedural or object-oriented programming language, or inassembly or machine language if desired; and in any case, the languagecan be a compiled or interpreted language. Suitable processors include,by way of example, both general and special purpose microprocessors.Generally, a processor will receive instructions and data from aread-only memory and/or a random access memory. The essential componentsof a computer are a processor for executing instructions and a memory.Generally, a computer will include one or more mass storage devices forstoring data files; such devices include magnetic disks, such asinternal hard disks and removable disks; magneto-optical disks; andoptical disks. Storage devices suitable for tangibly embodying computerprogram instructions and data include all forms of non-volatile memory,including by way of example semiconductor memory devices, such as EPROM,EEPROM, and flash memory devices; magnetic disks such as internal harddisks and removable disks; magneto-optical disks; and CD-ROM disks. Anyof the foregoing can be supplemented by, or incorporated in, ASICs(application-specific integrated circuits).

To provide for interaction with a user, the invention can be implementedon a computer system having a display device such as a monitor or LCDscreen for displaying information to the user and a keyboard and apointing device such as a mouse or a trackball by which the user canprovide input to the computer system. The computer system can beprogrammed to provide a graphical user interface through which computerprograms interact with users.

The invention has been described in terms of particular embodiments.Other embodiments are within the scope of the following claims. Forexample, steps of the invention can be performed in a different orderand still achieve desirable results. Feature vectors or information formultiple reference objects can be combined in a single step, rather thanin a series of pair wise combinations. Information for media objects andreference objects can be expressed as models or high-dimensionalmatrices. Non-Euclidian functions other than Min and Max can be used tocombine or compare feature vectors.

Having thus described the invention, what is claimed is:
 1. Acomputer-implemented method comprising: receiving identification of twoor more reference video objects; calculating, by a computer, videoparameters for each reference video object, wherein the video parameterscharacterize image features and audio features in each reference videoobject that are common to a first reference video object and at least asecond reference video object; combining, by the computer, the videoparameters for the first reference video object and at least the secondreference video object to generate composite reference information; andcomparing, by the computer, the composite reference information to videoobjects in a collection to identify one or more video objects in thecollection having features described by the composite referenceinformation.
 2. The computer-implemented method of claim 1, wherein theimage features comprise texture, color, shape, or combinations thereof.3. The computer-implemented method of claim 1, wherein the audiofeatures comprise Fourier transform information, wavelet decompositioninformation, fractal measures, or combinations thereof.
 4. Thecomputer-implemented method of claim 1, wherein the video parametersinclude fractal measures derived from an analysis of combined audio andvisual aspects of the video object.
 5. The computer-implemented methodof claim 1, wherein the video parameters are combined to encompasscharacteristics of any of the reference video objects.
 6. Thecomputer-implemented method of claim 1, wherein the video parameters arecombined to specify characteristics that are common to all of thereference video objects.
 7. The computer-implemented method of claim 1,further comprising receiving identification of a new reference objectafter identifying one or more video objects in the collection havingfeatures described by the composite reference information.
 8. Thecomputer-implemented method of claim 7, further comprising modifying thecomposite reference information based on the new reference object. 9.The computer-implemented method of claim 8, wherein the new referenceobject is an audio object, an image object, a text object, or a videoobject.
 10. The computer-implemented method of claim 1, wherein thecalculated video parameters are combined into a feature vector for eachreference video object.
 11. The computer-implemented method of claim 10,wherein the feature vector includes multiple sets of image componentsand audio components.
 12. The computer-implemented method of claim 10,wherein the feature vector is standardized to sum to unity.
 13. Thecomputer-implemented method of claim 10, wherein the feature vector isnot standardized to sum to unity, and comparing comprises using amodified Min function to produce vectors for which a sum of componentsranges from 0 to
 1. 14. A system comprising: a processor configured toexecute instructions; a memory; and computer program instructionsembodied on the memory, which, when executed by the processor, cause theprocessor to: receive identification of two or more reference videoobjects; calculate video parameters for each reference video object,wherein the video parameters characterize image features and audiofeatures in each reference video object that are common to a firstreference video object and at least a second reference video object;combine the video parameters for the first reference video object and atleast the second reference video object to generate composite referenceinformation; and compare the composite reference information to videoobjects in a collection to identify one or more video objects in thecollection having features described by the composite referenceinformation.
 15. The system of claim 14, wherein the audio featurescomprise Fourier transform information, wavelet decompositioninformation, fractal measures, or combinations thereof.
 16. The systemof claim 14, wherein the video parameters include fractal measuresderived from the analysis of combined audio and visual aspects of thevideo object.
 17. The system of claim 14, wherein the calculated videoparameters are combined into a feature vector for each reference videoobject.
 18. The system of claim 17, wherein the feature vector includesmultiple sets of image components and audio components.
 19. The systemof claim 17, wherein the feature vector is standardized to sum to unity,or, if the feature vector is not standardized to sum to unity, using amodified Min function to produce vectors for which a sum of componentsranges from 0 to
 1. 20. Computer program instructions, encoded on one ormore computer-readable memories, which, when executed by a computerprocessor, cause the processor to: receive identification of two or morereference video objects; calculate video parameters for each referencevideo object, wherein the video parameters characterize image featuresand audio features in each reference video object that are common to afirst reference video object and at least a second reference videoobject; combine the video parameters for the first reference videoobject and at least the second reference video object to generatecomposite reference information; and compare the composite referenceinformation to video objects in a collection to identify one or morevideo objects in the collection having features described by thecomposite reference information.